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Image processing method and apparatus. 

(57) An image processing apparatus comprises: 
an image input unit to Input an image ; an 
image thinner for getting the OR of (m dots in 
the vertical direction) x (n dots in the horizontal 
direction) for the input image from the image 
input unit, thereby thinning out the (m x n) 
pixels and newly forming one pixel ; a labeler 
for adding labels every line to black pixels of the 
image which was thinned out by the image 
thinner and adding the same label to the pixels 
which are continuous in the vertical, horizontal, 
or oblique direction and simultaneously tracing 
a rectangle ; and a separator detector for dis- 
tinguishing a rectangle con-esponding to a 
sentence portion, a rectangle corresponding to 
a figure or photograph, a rectangle correspond- 
ing to a table, a rectangle corresponding to a 
separator, and the like by using a width, a 
height, an area, and a pixel density (the number 
of pixels to the area) of the rectangle added with 
the label by the labeler On the basis of the 
result of the detection of the separator detector, 
the area of the input image is divided. 
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BACKGROUND OF THE INVENTION 

Field of the Invention 

The invention relates to an image processing ap- 
paratus for, particularly, dividing an input image into 
a character area and an area of a figure, table, or the 
like in an electronic apparatus such as OCR (optical 
character recognizing) apparatus, copying apparatus, 
facsimile apparatus, or the like. 

Related Background Art 

In a conventional image processing apparatus, 
there is an apparatus using a run length analyzing 
method as area dividing means whereby a distribu- 
tion of run lengths is examined and a character area, 
a figure area, or the like is divided in accordance with 
a length of white run or black run, an apparatus using 
a spectrum analyzing method whereby a Fourier 
spectrum of an input image is analyzed and the input 
image is divided into various kinds of areas, an appa- 
ratus using a projection analyzing method whereby 
projections (histograms) In the vertical and horizontal 
directions are alternately repeatedly obtained and the 
area is divided from the information of a peripheral 
portion as disclosed in JP-A-1 -15889. 

The above conventional techniques, however, 
have problems such that it takes an extremely long 
time for arithmetic operating processes and process- 
es are executed to the pixels of an image, so that a 
memory area remarkably increases, a dividing preci- 
sion of a complicated area construction to the image 
Is low, and the like. 

In the conventional image processing apparatus, 
as area dividing means, there is used a method 
whereby a discrimination regarding whether an area 
is a table or another area is made on the basis of a 
ratio of the area of circumscribed rectangle and the 
number of pixels, and the area is divided on the basis 
of the result of the discrimination. 

The above conventional technique, however, has 
a problem such that in the case where a separator or 
the like (modified separator) formed by complicated 
polygonal lines or the like is handled as one area, 
when the number of pixels for the area of the circum- 
scribed rectangle increases, such an area is errone- 
ously recognized as a table area and a dividing pre- 
cision of the area decreases. 

In the conventional image processing apparatus, 
in the case where the attributes (character, figure, ta- 
ble, etc.) of the divided area is erroneously discrimin- 
ated, the area data which was erroneously discrimin- 
ated is eliminated and a correct area frame is newly 
designated by a manual operation of the operator. 

In a conventional technique such that the correc- 
tion of the divided area is executed by a manual op- 
eration of the operator, in the case where a plurality 



of erroneous divided-areas exist, there is a problem 
such that a burden on the operator increases and a 
long time is needed as a ratio to the processing time 
of the whole image processes. 

5 In such a kind of well-known image processing 

apparatuses, in the case where the image area was 
erroneously divided, area data of the erroneous area 
is eliminated and the operator newly designates the 
correct area frame by using a pointing device or the 

10 like. 

The above conventional technique, however, has 
problems such that in case of a complicated area con- 
struction, a burden of the operation of the operator in- 
creases, it takes an extremely long time as a ratio to 
15 the processing time of the whole image processes, 
and the like. 

SUMMARY OF THE INVENTION 

20 In consideration of the above problems, an enrv 

bodimentof the invention provides an image process- 
ing apparatus which can realize the reduction of the 
processing time when an area is divided, the de- 
crease in size of a recording area, the improvement 

25 of a dividing precision of a complicated area construc- 
tion, and the like. After an input image was thinned 
out, it is further converted into rectangle data by using 
a labeling and an area is divided into a character area 
and an area of a figure or photograph, a table, a sep- 

30 arator, an index,.or the like. Therefore, there are ef- 
fects such that without changing the existing elec- 
tronic parts, circuits, and the like, (1) reduction in 
processing time, (2) decrease in size of a memory 
area or the like, (3) improvement of a dividing preci- 

35 sion of a complicated area construction, and the like 
are obtained. 

According to the invention, the input image is 
converted into the rectangle data and is classified 
into a character area and an area of a figure, photo- 

40 graph, table, separator, and the like and, further, par- 
tial histograms of the left and right edges in the vert- 
ical and horizontal directions of the area which was 
temporarily determined to be a table area are calcu- 
lated and compared, thereby deciding whether the 

45 area is a table area or not. There are, consequently, 
effects such that without changing the existing elec- 
tronic parts, circuits, or the like, (1) the improvement 
of the precision or the like is obtained for the division 
of a complicated area construction, particularly, the 

50 division of an area construction including a table, (2) 
the processing time is short because the histograms 
are partially obtained, and the like. 

According to the invention, an input image is con- 
verted into rectangle data and is classified into a char- 

55 acter area and an area of a figure or a photograph, ta- 
ble, separator, or the like, and further the rules in the 
vertical and horizontal directions of the area which 
was temporarily decided as a table area are detected 
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and counted, thereby deciding whether the area is a 
table area or not. Thus, there is an effect such that 
without changing the existing electronic parts, cir- 
cuits, or the like, (1) the improvement of the precision 
or the like is obtained for the division of a complicated 
area construction, particularly, the division of an area 
construction including a table. 

According to the invention, an input Image is con- 
verted into rectangle data and Is classified into a char- 
acter area or an area of a figure or a photograph, ta- 
ble, separator, or the like, and a ratio of the area in a 
frame to the area of the area which was temporarily 
decided as a table area Is obtained. When such a ratio 
is equal to or larger than a threshold value, the area 
is determined to be a table. Thus, there are effects 
such that without changing the existing electronic 
parts, circuits, and the like, 

(1) the improvement of the precision or the like is 
obtained for the division of a complicated area 
construction, particularly, the division of an area 
construction including a table, 

(2) since the area in the frame Is used, even when 
the table frame slightly lacks, the decision about 
the table can be accurately performed. 

(3) since the area in the frame is used, the deci- 
sion about the table is strong for an inclination 
and can be accurately performed, 

(4) and the tike. 

According to the invention, by combining the er- 
roneously divided areas and correcting the errone- 
ously presumed attributes, there are effects such that 
without changing the existing area dividing process, 
(1) an operating efficiency can be improved, (2) a 
processing time can be reduced, (3) and the like. 

It is another object of the invention to provide an 
Image processing apparatus which can combine er- 
roneously divided areas by a simple operation and the 
attributes of each area can be also easily corrected. 

By providing correcting means for integrated I y 
combining special divided areas, the erroneously div- 
ided areas are combined and, further, the necessary 
area attributes can be corrected. Therefore, without 
changing the existing area dividing process. (1) an 
operating efficiency can be improved. (2) a process- 
ing time can be reduced. (3) and the like. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 Is a block diagram showing a schematic 
construction of an image processing apparatus of 
an embodiment of the invention; 
Fig. 2 is a flowchart showing a whole image proc- 
essing procedure for the area division of an em- 
bodiment of the invention; 
Fig. 3 is an explanatory diagram showing an ex- 
ample of a labeling process of an embodiment of 
the invention; 

Fig. 4 is an explanatory diagram showing a struc- 
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ture of rectangle data in a memory unit of an em- 
bodiment of the invention; 
Fig. 5 is comprised of Figs. 5A and 5B showing 
flowcharts of a processing procedure of a sepa- 

5 rator detector of an embodiment of the Invention; 

Fig. 6 is an explanatory diagram showing cutting 
processes based on a density D and an area S of 
the attributes of a rectangle (area) of an embodi- 
ment of the invention; 

10 Fig. 7 is a flowchart showing a processing proce- 

dure of a writing direction detector of an embodi- 
ment of the invention; 

Fig, 8 Is an explanatory diagram showing an ex- 
ample of a detecting process of a paragraph of an 
15 embodiment of the invention; 

Fig. 9 is a flowchart showing a processing proce- 
dure of an index detector of an embodiment of the 
Invention; 

Fig. 10 is an explanatory diagram showing an ex- 
20 ample of a combining process of rectangles of an 

embodiment of the invention; 
Fig. 11 is an explanatory diagram showing an ex- 
ample of a combining process of separator rec- 
tangles of another embodiment of the invention; 
25 Fig. 12 Is a block diagram showing a schematic 

construction of an image processing apparatus of 
an embodiment 2; 

Fig. 13 is a flowchart showing an image process 
In the image processing apparatus of the embodi- 

30 ment 2; 

Fig. 1 4 is a diagram showing an example of a lat>- 
eling process of the embodiment 2; 
Fig. 15 is a diagram showing a rectangle data 
structure and the relation between the rectangle 

35 label and the attributes of the embodiment 2; 

Fig. 16 is comprised of Figs. 16Aand 16B show- 
ing flowcharts of a process of a separator detec- 
tor of the embodiment 2; 

Fig. 17 is a diagram showing cutting processes 
40 based on a density D and an area S of the attri- 

bute of a rectangle (area) of the embodiment 2; 
Fig. 1 8 is a diagram showing a histogram when an 
area is determined to be a table area of the em- 
bodiment 2; 

45 Fig. 1 9 is a diagram showing a histogram when an 

area is determined to be a modified separator 
area of the embodiment 2; 
Fig. 20 is a flowchart showing an example of 
processes of a table area determiner of the em- 

50 bodiment 2; 

Fig. 21 is a-diagram showing a histogram when 
an area is determined to be a modified separator 
area of the embodiment 2; 
Fig. 22 is a flowchart showing an example of 

55 processes of the table area determlnerof the em- 

bodiment 2; 

Fig. 23 is a flowchart showing an image process 
in the image processing apparatus of the embodi- 
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ment 2; 

Fig. 24 is a diagram showing a histogram when an 
area is determined to be a table area of the em- 
bodiment 2; 

Fig. 25 is a diagram showing a histogram when an 5 
area is determined to be a modified separator 
area of the embodiment 2; 
Fig. 26 is a flowchart showing an example of 
processes of a table rule detector and the table 
area determiner of the embodiment 2; io 
Fig. 27 is a flowchart showing an example of 
processes of the table area determiner of the em- 
bodiment 2; 

Fig. 28 is a flowchart showing an example of 
processes of the table area determiner of the em- 15 
bodiment 2; 

Fig. 29 is a diagram showing a table and rectan- 
gles showing areas of the table of the embodi- 
ment 2; 

Fig. 30 is a-diagram showing the inside of a table 20 
and rectangles showing areas of the table of the 
embodiment 2; 

Fig. 31 is a diagram showing a modified separator 
and rectangles showing an area of the modified 
separator of the embodiment 2; 25 
Fig. 32 is a diagram showing the inside of a modi- 
fied separator and rectangles showing an area of 
the modified separator of the embodiment 2; 
Fig. 33 is a flowchart showing an example of 
processes of a table area determiner of the em- 30 
bodiment 2; 

Fig. 34 is a diagram showing an area of a table 
and each pixel of the embodiment 2; 
Fig. 35 is a block diagram showing a schematic 
construction of the image processing apparatus 35 
of the embodiment 2; 

Fig. 36 is a flowchart showing an area division 
correcting process of the embodiment 2; 
Fig. 37 is a diagram showing an example of the 
area division correcting process of the embodi- 40 
ment 2; 

Fig. 38 is a diagram showing an area data struc- 
ture of the embodiment 2; 
Fig. 39 is a flowchart showing an area attribute 
correcting process of the embodiment 2; 45 
Fig. 40 is a diagram showing an example of an 
area attribute correcting process display unit of 
the embodiment 2; 

Fig. 41 is a block diagram showing a schematic 

construction of an image processing apparatus so 

according to an embodiment 3; 

Fig. 42 is a flowchart showing an area division 

correcting process of the embodiment 3; 

Fig. 43 is a diagram showing an example of the 

area division correcting process of the embodi- 55 

ment 3; 

Fig. 44 is a diagram showing an area data struc- 
ture of the embodiment 3; 



Fig. 45 is a flowchart showing an area attribute 
correcting process of the embodiment 3; and 
Fig. 46 is a diagram showing an example of a dis- 
play unit of the area attribute correcting process 
of the embodiment 3. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

(Embodiment 1) 

The first embodiment of the invention will now be 
described hereinbelow withreference to Figs. 1 to 11. 

Fig. 1 shows a schematic construction of an im- 
age processing apparatus of an embodiment of the in- 
vention. In the diagram, reference numeral 101 de- 
notes an input unit of image data; 102 a central proc- 
essing unit (CPU) to control the apparatus and exe- 
cute an arithmetic operation of a process in each unit; 
103 a memory to store control programs of flow- 
charts, which will be explained hereinlater. and vari- 
ous kinds of data; 104 an output unit to generate the 
results of the operating processes, the results of the 
image processes, and the image data; and 105 an 
area divider to execute a whole area dividing process. 

An internal construction of the area divider 105 
will now be described. First, reference numeral 106 
denotes an image thinner to thin out an input image 
in a manner such that (m x n) pixels are thinned out 
by getting the OR of the vertical (m) dots and the hor- 
izontal (n) dots of the input image and one pixel is 
newly formed. Reference numeral 1 07 denotes a lat>- 
eler for labeling the pixel obtained by the thinning-out 
process and for simultaneously forming initial rectan- 
gle data. That is, a label is added to the black pixels 
of the image for every line and the same label is add- 
ed to the pixels which are continuous in the vertical, 
lateral, or oblique direction and, at the same time, a 
rectangle is traced. Reference numeral 108 denotes 
a separator detector to detect a separator, a table, a 
figure, or the like. By using the width of the above rec- 
tangle, height, area, and the number of pixels, name- 
ly, pixel density to the area, the separator detector 
108 distinguishes a rectangle corresponding to a 
character portion (body), a rectangle corresponding 
to a figure or photograph, a rectangle corresponding 
to a table, a rectangle corresponding to a separator, 
and the like. Reference numeral 109 denotes a writing 
direction detector to detect the writing direction of a 
sentence such as vertical writing, horizontal vwiting, 
or the like. The writing direction detector 109 pre- 
sumes a vertical writing sentence or a horizontal writ- 
ing sentence by comparing the width and height of 
the rectangle corresponding to the character portion 
(body). Reference numeral 110 denotes an index de- 
tector to detect an index by using the writing direction 
and the character size. Reference numeral 111 de- 
notes a rectangle combiner to combine rectangles by 
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using an imaginary rectangle which is obtained by en- 
larging the size of Inherent rectangle. 

A flowchart of Fig. 2 shows an Image processing 
procedure in the area divider 105 in Fig. 1. A control 
program for executing the above image process has 
been stored In the memory 103. 

(1) step S201 

An original image is Inputted from the image input 
unit 101. 

(2) step S202 

In the Image thinner 106, the OR of the vertical 
(m) dots and the horizontal (n) dots of the original im- 
age is calculated and the (m x n) pixels are thinned 
out, thereby newly forming one pixel. When a black 
pixel of at least one dot exists in the (m x n) pixels of 
the original Image, the thinned-out pixel is set to 
black. 

(3) step S203 

In the labeler 107, a label is added to the black 
pixel of the thinned-out pixel for every line and the 
same label is added to the pixels which are continu- 
ous in the vertical, lateral, or oblique direction and, at 
the same time, the rectangle is traced. 

When explaining Fig. 3 as an example, a label 1 
is added to the pixel (A) which is first detected. The 
coordinates (Xa, Ya) of the pixel A are set to an initial 
point and a terminal point of the rectangle, the num- 
ber of pixels is set to 1, the same label 1 as that of the 
pixel is added to a rectangle label to distinguish the 
rectangle, and the above data is stored as rectangle 
data into the memory 103 as shown in Fig. 4. 

A label 2 added to a pixel B in which there is no 
continuous pixel in the left direction (since the pixel 
B is located on the first line, there is also obviously 
no continuous pixel from the upper line). The coordin- 
ates (Xb, Yb) of the pixel B are set to an initial point 
and a terminal point of the rectangle, the number of 
pixels is set to 1 . t he same label 2 as t hat of t he pixel 
is added to the rectangle label to distinguish the rec- 
tangle, and the above data is also stored as rectangle 
data into the memory 103 as shown in Fig. 4. 

After the labeling of the first line was finished as 
mentioned above, the processes progress to the sec- 
ond line. 

Since the first pixel C of the second line is con- 
tinuous with the pixel A of the label 1 from the upper 
line, the pixel label 1 is added to the pixel C. "1" is add- 
ed to the number of pixels for the rectangle data of t he 
rectangle label 1, so that the total number of pixels is 
equal to 2, the rectangle label is unchanged and is 
maintained to 1, and only the terminal point of the rec- 
tangle coordinates is updated from (Xa, Ya) to (Xa, 



Yc) (the coordinate of the initial point is unchanged). 

Since the next pixel D is continuous with the pixel 
C from the left, the label 1 is added to the pixel D. "1" 
is added to the number of pixels for the rectangle data 

5 of the rectangle label 1, so that the total number of 
pixels is set to 3. The rectangle label is maintained to 
1 without being changed. Only the terminal point of 
the rectangle coordinates is updated from (Xa, Yc) to 
(Xd, Yc) (the Y coordinate of the terminal point is un- 

10 changed). In this instance, both of the pixels D and B 
are obliquely continuous and are continuous with the 
pixel C, so that the label of the pixel B is changed from 
the label 2 to the label 1. The number of pixels of the 
rectangle label 2 is added to that of the rectangle data 

15 of the rectangle label 1, so that the total number of 
pixels is set to 4. The rectangle label is maintained to 
1 without being changed. Only the terminal point of 
the rectangle coordinates is updated from (Xd, Yc) to 
(Xb, Yd) so as to include all of the pixels A, B, C, and 

20 D. With respect to the rectangle data of the rectangle 
label 2, it is invalidated by setting the rectangle label 
into 0. 

After the labeling of the second line was finished 
as mentioned above, the processes progress to the 

25 third line. 

Since the first pixel E of the third line is obliquely 
continuous with the pixel C, the pixel label 1 is added 
to the pixel E. "I" is added to the number of pixels for 
the rectangle data of the rectangle label 1, so that the 

30 total number of pixels is set to 5. The rectangle label 
is maintained to 1 without being changed. As for the 
rectangle coordinates, the initial point is updated from 
(Xa, Ya) to (Xe, Ya) and the terminal point is updated 
from (Xb, Yd) to (Xb, Ye). Namely, in the rectangle 

35 data in Fig. 4 in the memory 1 03 in this instance, the 
rectangle label is set to 1 , the initial point coordinates 
are set to (Xe, Ya), the terminal point coordinates are 
set to (Xb, Ye), the pixel label is set to 1 , and the num- 
ber of pixels is set to 5. 

40 In a manner similar to the above, the labeling is 

executed to all of the thinned-out pixels and the trac- 
ing of the rectangle is performed. 

(4) step 8204 

45 

After completion of the above labeling and trac- 
ing of the rectangle, the separator detector 108 dis- 
tinguishes a rectangle corresponding to the character 
portion (body), a rectangle corresponding to the f ig- 

50 ure, photograph, table, or the like, a rectangle corre- 
sponding to the separator, and the like by using the 
width W of the rectangle, height H, area S, and the 
number of pixels for the area, namely, the pixel den- 
sity D (which can be easily obtained by calculating us- 

55 ing those rectangle data). The above processes will 
now be described in detail hereinbelow with reference 
to flowcharts of Figs. 5A and 5B. 

With respect to the separator, when the width W 
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is equal to or less than a threshold value TwI and the 
height H is equal to or larger than the value which is 
twice as large as a threshold value Tw2 of the width 
W (step S503) or when the width W is larger than the 
threshold value TwI and the height H is equal to or 
larger than the value which is three times as large as 
a threshold value Tw3 of the width W (step S504), the 
separator Is regarded as a separator which is vertical- 
ly long and is unified by setting the rectangle label in 
the memory 103 into -3 and the pixel label construct- 
ing the rectangle keeps the current label number sep- 
arately from the rectangle label (step S514). 

A rectangle in which the above width and height 
were replaced is also judged in a manner similar to the 
above (steps S505, S506). If YES in each of the steps 

5505 and S506, the separator is regarded as a sep- 
arator which is horizontally long and the rectangle lab- 
el is changed to -3. 

As shown in Fig. 6, if NO in all of steps S503 to 

5506 and when the pixel density D is equal to or less 
than a threshold value Tdl (step S507), the separator 
is regarded as a modified separator such as a key- 
shape or the like and the rectangle label is set to -3 
(step S514). 

If NO in step S507 and when the area S is larger 
than a threshold value Tsl (step S508), so long as the 
pixel density D is less than a threshold value Td2 
(step S509), the rectangle is regarded as a table and 
the rectangle label is changed to -4 (step S515). 
When the pixel density D is equal to or larger than the 
threshold value Td2, the rectangle is regarded as a 
figure or photograph and the rectangle label is 
changed to -5 (step S516). 

A rectangle in which the pixel density D is equal 
to or larger than a threshold value TdS (step S511} 
when the area S is equal to or less than the threshold 
value Ts1 and is equal to or larger than a threshold 
value Ts2 (step S510) or a rectangle in which both of 
the width W and the height H are equal to or larger 
than a threshold value Tw4 and the pixel density D is 
equal to or larger than a threshold value TdS (step 
S513) is also regarded as a figure or photograph and 
the rectangle label is changed to -5 (step S516). 

Further, a rectangle in which the pixel density D 
is less than a threshold value Td4 (step S512) when 
the area S is equal to or less than the threshold value 
Ts1 and is equal to or larger than the threshold value 
Ts2 (step S510) is regarded as a table and the rectan- 
gle label is changed to -4 (step S515). 

As mentioned above, a rectangle corresponding 
to a figure, photograph, table, or the like, a rectangle 
corresponding to a separator, or the like is detected, 
the remaining rectangle is regarded as a body (char- 
acter portion), and the rectangle label is unchanged 
and is set to the same label as the pixel label (step 
S517). Fig. 6 shows the relation between the above 
threshold values in case of the area S and pixel den- 
sity D and the classification of the separator or the 
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like. 

(5) step S205 

5 In the writing direction detector 109 detects the 

writing direction of a sentence such as vertical writing, 
horizontal writing, or the like. The writing direction de- 
tecting process will now be described with reference 
to a flowchart of Fig. 7. 

10 In case of a horizontal writing sentence, the rec- 

tangle remaining as a body easily becomes a rectan- 
gle which is laterally long because the pixels which 
were thinned out in the lateral direction are continu- 
ous. In case of a vertical writing sentence, the rectan- 

15 gle remaining as a body easily becomes a rectangle 
which is vertically long. Therefore, in the writing direc- 
tion detector 109, the average values (w and h) of the 
width W and height H of the rectangle of the body are 
calculated (step S701). When the average width w is 

20 larger than the average height h, the sentence is re- 
garded as a sentence having a large amount of hori- 
zontal writing sentences and the average height h is 
set to a character size of one character (steps S702, 
S703). On the contrary, when the average height h is 

25 larger than the average width w, the sentence is re- 
garded as a sentence having a large amount of vert- 
ical writing sentences and the average width w is set 
to a character size of one character (steps S702, 
S710). 

30 As shown in Fig. 8, a histogram of rectangles is 

obtained in the direction opposite to the writing direc- 
tion (steps S704, S711). A position below a threshold 
value Tk is set to a separation of a paragraph from the 
shape of a peripheral distribution (steps S705, S712). 

35 A histogram of the rectangles is obtained every para- 
graph in the same direction as the writing direction 
(steps S708, S715). Alength of continuous black pix- 
els is set to a character size within the paragraph from 
the shape of the peripheral distribution and a length 

40 of continuous white pixels is detected as a line pitch 
(steps S709, S716). The above detecting process is 
repeated until the end of the paragraph (steps S706, 
S713). 

45 (6) Step S206 

In the index detector 110, an index is detected 
from the writing direction and the character size. As 
shown in a flowchart of Fig. 9, when considering the 

50 case of the vertical writing as an example, a rectangle 
whose width W is equal to or larger than the value of 
. a t hreshold value Tml X the character size w is detect- 
ed as a candidate of the index rectangle from the 
body rectangle (step S905). Further, when the height 

55 H of rectangle is smaller than the value of (a threshold 
value Tm2) times of the character size w (step S906), 
a sentence is regarded as a sentence In which the 
horizontal written characters are continuous and such 

6 



11 



EP 0 587 450 A1 



12 



a sentence is regarded as an index (step S908). 

Since there Is a case where a rectangle in which 
characters of a body were combined is also Included 
In the rectangles which were regarded as indices from 
the size of rectangle as mentioned above, with re- 
spect to the rectangle in which the distance between 
the body rectangle and the index rectangle Is closer 
than a threshold value T1 (step S907). it is corrected 
from the index to the body (step S909). 

Among the body rectangles remaining by the 
above processes, a rectangle such that none of the 
rectangles of the body and index exists within a range 
of the character size w is set into an index as an iso- 
lated rectangle (steps S910 to S913). 

(7) step S207 

In the rectangle combiner 111, the rectangles 
corresponding to the bodies which exist at random 
without any relation and the rectangles of the Indices 
are combined. 

An example Is shown In Fig. 10. The combination 
of rectangles A1 and B will now be considered. An im- 
aginary rectangle A1' which is obtained by enlarging 
the rectangle A1 by only Px in the X direction and by 
only Py in the Y direction is now considered. The per- 
iphery of the rectangle A1' is searched to see if there 
is a rectangle which is come Into contact with or is in- 
scribed in the rectangle AV or not. When a rectangle 
which is in contact with A1' exists like a rectangle B, 
by combining the rectangles A1 and B. a combined 
rectangle A2 Is newly formed. By updating the rectan- 
gle data of the rectangle A1 , the updated data is stor- 
ed and the data of the rectangle B is invalidated. In 
this Instance, the values of Px and Py are obtained 
from the character size and line pitch. In the case 
where the rectangle formed by combining the rectan- 
gles is come into contact with a figure, table, separa- 
tor, or the like, the combining process itself is Invali- 
dated and the original rectangles are maintained as 
they are. 

As mentioned above, the indices are first com- 
bined and the body rectangles are subsequently com- 
bined. In this instance, in case of combining the bod- 
ies, it is prevented that the rectangle Is come Into con- 
tact with not only a figure, table, or separator but also 
an index. When the rectangle is come into contact 
with the index, the combining process itself is invali- 
dated and the original rectangles are maintained as 
they are. 

(8) step S208 

Finally, the rectangle data of various kinds of 
areas obtained as mentioned above is generated from 
the output unit 104 to the outside together with the 
image data. 

In the image inputting process in step S201 in Fig. 
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2 mentioned above, by judging whether the Input im- 
age has multivalues or not and by converting the input 
image into a binary image, the area dividing process 
can be also executed even when the input image is a 

5 multivalue image such as a color image or the like. 

In the image thinning process in step S202, when 
the number of pixels of the input image is sufficiently 
small to be a value which doesn't decrease the proc- 
essing speed, such an image thinning process can be 

10 also omitted. 

In the detection of a separator or the like in step 

5204, by distinguishing the rectangle label In depend- 
ence on a difference between the vertical and lateral 
directions of the separator, a difference between the 

15 threshold values when obtaining a figure, table, orthe 
like, etc., the further fine area attributions can be 
classified. 

Likewise, in the detection of a separator or the 
like in step S204, tn the case where only an image of 
20 a special attribute is Inputted tirom the beginning like 
only a sentence or the like, the detecting process of 
a separator or the like can be also omitted. 

In the detection of the writing direction in step 

5205, in place of obtaining the average width and 
25 average height, by obtaining the most frequent val- 
ues, those values can be also used. 

Similarly, in the detection of the writing direction 
in step S205, by again confirming the writing direction 
every paragraph, the processes can be also accu- 

30 rately executed to an original in which the vertical 
writing sentences and the horizontal writing sentenc- 
es mixedly exist. 

In the detection of the writing direction in step 
8205, when only the sentences In a predetermined 

35 writing direction are inputted from the beginning, by 
setting the writing direction to a fixed value, such a 
detecting process can be replaced. 

In the index detection in step 8206, when only the 
image having no index is inputted from the beginning, 

40 such a detecting process can be also omitted. 

In the combining process of the rectangles in step 
8207, as shown In Fig. 11, when a rectangle is come 
into contact with a rectangle such as a separator or 
the like, a check is further made to see If the rectangle 

45 is also In contact with the actually thinned-out pixel or 
not, and when the rectangle is not in contact with such 
a pixel, by executing the combining process, the ap- 
paratus can also cope with a slanted image. 

Likewise, in the combining process of rectangles 

50 in step 8207, in case of an image such that the num- 
ber of pixels is so small that the image thinning proc- 
ess can be omitted, such a combining process can be 
also omitted. 

In the last outputting process In step 8208, only 

55 the image data of a necessary area can be also gen- 
erated with reference to the rectangle data of various 
kinds of areas, so that the memory area and the proc- 
essing time can be further reduced. 

7 
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(Embodiment 2) 

A preferred second embodiment of the invention 
will now be described with reference to Figs. 1 2 to 40. 

Fig. 12 is a block diagram showing an image proc- 
essing apparatus of the embodiment. 

In the diagram, reference numeral 101 denotes 
the input unit of image data. 

Reference numeral 102 denotes the central proc- 
essing unit (CPU) to control the apparatus and exe- 
cute an arithmetic operation of the process in each 
unit in accordance with the control programs stored 
in the memory 103. 

Reference numeral 103 denotes the memory to 
store the control programs as shown in flowcharts, 
which will be explained hereinlater, and various kinds 
of data. The memory 103 includes an ROM and an 
RAM. 

Reference numeral 104 denotes the output unit to 
generates the results of the arithmetic operating 
processes, the results of the image processes, and 
image data. 

Reference numeral 105 denotes the area divider 
showing the whole area dividing process. 

Reference numeral 107 denotes the labeler for 
adding a label to the pixels of the input image and for 
simultaneously forming initial rectangle data. 

Reference numeral 207 denotes an attribute de- 
tector to detect the attribute of a separator, a table, a 
figure, or the like. 

Reference numeral 208 denotes a table rule de- 
tector to detect straight lines constructing a table. 

Reference numeral 209 denotes a table area de- 
terminer to determine whether an area is a table area 
or not from a state of the table rules. 

Fig. 13 is a flowchart showing the image process 
in the image processing apparatus of the embodi- 
ment. The control programs to execute the image 
process have been stored in the memory 103. 

(step S201) 

The original image is first inputted from the image 
input unit 101. 

(step S202) 

In the labeler 107, the labels are added to the 
black pixels of the thinned-out image for every line 
and the same label is added to the pixels which are 
continuous in the vertical, lateral, and oblique direc- 
tions, thereby simultaneously tracing a rectangle. 

When explaining Fig. 14 as an example, the label 
1 is added to the pixel A which is first detected. The 
coordinates (Xa, Ya) of the pixel A are set to an initial 
point and a terminal point of the rectangle, the num- 
ber of pbcels is set to 1, the same label is as that of 
the pixels is added to the rectangle label to distin- 
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guish the rectangle, and the above data is stored into 
the memory as rectangle data (Fig. 15). 

Subsequently, the label 2 is added to the pixel B 
such that there is no continuous pixel in the left direc- 
5 tion (since the pixel B is located on the first line, there 
is also no continuous pixel from the upper line). The 
coordinates (Xb, Yb) of the pixel A are set to an initial 
point and a terminal point of the rectangle. The num- 
ber of pixels is set to 1 and the same label 2 as that 

10 of the pixels is added to the rectangle label to distin- 
guish the rectangle. The above data is also stored into 
the memory as rectangle data (Fig. 15). 

After completion of the labeling of the first line as 
mentioned above, the processes progress to the sec- 
ts ond line. 

Since the first pixel C of the second line is con- 
tinuous with the pixel A of the label 1 from the upper 
line, the pixel label 1 is added to the pixel C. "1 " is add- 
ed to the number of pixels for the rectangle data of the 

20 rectangle label 1 , so that the total number of pixels is 
set to 2. The rectangle label is maintained to 1 without 
being changed. Only the terminal point of the rectan- 
gle coordinates is updated from (Xa, Ya) to (Xa, Yc) 
(the coordinates of the initial point are unchanged). 

25 Since the next pixel D is continuous with the pixel 

C from the leftside, the label 1 is added. "1" is added 
to the number of pixels for the rectangle data of the 
rectangle label 1 , so that the total number of pixels is 
set to 3. The rectangle label is maintained to 1 without 

30 being changed. Only the terminal point of the rectan- 
gle coordinates is updated from (Xa, Yc) to (Xd, Yc) 
(the Y coordinate of the end point is unchanged). 

In this instance, the pixel D is also obliquely con- 
tinuous with the pixel B and continuously comes from 

35 the pixel C, so that the label of the pixel B is changed 
from the label 2 to the label 1. The number of pixels 
of the rectangle label 2 is added to that of the rectan- 
gle data of the rectangle label 1, so thatthetotal num- 
ber of pixels is set to 4. The rectangle label is main- 

40 tained to 1 without being changed. Only the terminal 
point of the rectangle coordinates is updated from 
(Xd, Yc) to (Xb, Yd) so as to include all of the pixels 
A, B, C, and D. The rectangle data of the rectangle 
label 2 is invalidated by setting the rectangle label into 

45 0. 

After completion of the labeling for the second 
line as mentioned above, the processes progress to 
the third line. 

Since the first pixel E of the third line is obliquely 
50 continuous with the pixel C, the pbcel label 1 is added 
and "1" is added to the number of pixels for the rec- 
tangle data of the rectangle label 1, so that the total 
number of pixels is set to five. The rectangle label is 
maintained to 1 without being changed. The initial 
55 point of the rectangle coordinates is updated from 
(Xa, Ya) to (Xe, Ya) and the terminal point is updated 
from (Xb, Yd) to (Xb. Ye). 

In a manner similar to the above, the labeling and 
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tracing of the rectangle are executed for all of the thin- 
ned-out pixels. 

(step S203) 

5 

In the attribute detector 207, after completion of 
the labeling and the tracing of the rectangle, a rectan- 
gle corresponding to the body, a rectangle corre- 
sponding to a figure, photograph, table, or the like, a 
rectangle corresponding to a separator, and the like 10 
are distinguished by using the width W of the rectan- 
gle, height H, area S. and the number of pixels for the 
area, namely, the pixel density D (those data can be 
easily obtained by calculating the rectangle data). 

When the width W is equal to or less than the is 
threshold value Twi and the height H is equal to or 
larger than (the threshold value Tw2) times of the 
width W (S503 in Fig. ISA), or when the width W is 
larger than the threshold value TwI and the height H 
is equal to or larger than (the threshold value Tw3) 20 
times of the width W (S504 in Fig. 16B), the separator 
Is regarded as a separator which is vertically long and 
Is unified by setting the rectangle label to -3, and the 
pixel labels constructing the rectangle is maintained 
as it is separately from the rectangle label (S514 in 25 
Fig. 16B). 

In a manner similar to the above separator in 
which the width and height were replaced (S505 and 
S506 in Fig. 1 6A), the separator is regarded as a sep- 
arator which is horizontally long and the rectangle lab- 30 
el is changed to -3. 

As shown in Fig. 17, when the pixel density D is 
equal to or less than the threshold value Tdl (S507 in 
Fig. 16B), the separator is regarded as a modified 
separator such as key-shaped or the like and the rec- 35 
tangle label is changed to -3 (S514 in Fig. 16B). 

When the area S is larger than the threshold val- 
ue Tsl (S508 in Fig. 16B), so long as the pexel density 
D is less than the threshold value Td2 (S509 in Fig. 
1 6B), the area is regarded as a table and the rectan- 40 
gle label is changed to -4 (S515 in Fig. 16B). When 
the pexel density D is equal to or larger than the 
threshold value Td2. the rectangle is regarded as a 
figure or photograph and the rectangle label is 
changed to -5 (S516 in Fig. 168). 45 

A rectangle when the pixel density D is equal to 
or larger than the threshold value Td3 (S511 in Fig. 
16) when the area S is equal to or less than the 
threshold value Tsl and is equal to or larger than the 
threshold value Ts2 (S510 in Fig. 16B) or a rectangle 50 
when the width W and height H are equal to or larger 
than the threshold value Tw4 and the pixel density D 
is larger than Td5 (S513 in Fig. 16B) is also regarded 
as a figure or a photograph, and the rectangle label 
is changed to -5 (S515 in Fig. 16B). 55 

Further, when the area S is equal to or less than 
the threshold value Tsl and is equal to or larger than 
the threshold value Ts2 (S510 in Fig. 16B), a rectan- 



gle in which the pixel density D is less than a thresh- 
old value Td4 (S512 in Fig. 1 SB) is regarded as a table 
and the rectangle label is changed to -4 (S51 5 in Fig. 
16B). 

In a manner similar to the above, a rectangle cor- 
responding to a figure, photograph, table, or the tike, 
a rectangle correspond to the separator, and the like 
are detected, and the remaining rectangles are re- 
garded as bodies and the rectangle label is main- 
tained and is set to the same pixel label (S517 in Fig. 
16B). 

(step S204) 

Subsequently, in the table area determiner 208, 
a width when calculating a histogram is decided from 
the width W and height H of the rectangle which was 
determined to be a table area (S901 in Fig. 20). 

Partial histograms of the left and right edges in 
the vertical and horizontal directions are calculated 
(S902 in Fig. 20) foronly the pixels (701 in Fig. 18, 801 
In Fig. 1 9) (they can be easily obtained by comparing 
and selecting the pixel label of the rectangle data and 
the pixel label of each pixel in the rectangle) corre- 
sponding to the table firame excluding the bodies (702 
in Fig. 18, 802 in Fig. 19) or the like in the rectangle 
which was determined to be a table area. A threshold 
value Tkh of a histogram regarding whether a line is 
a table frame line which is parallel with the Y axis or 
not firom the height of the rectangle and a threshold 
value Tkw of a histogram regarding whether a line is 
a table frame line which is parallel with the X axis or 
not from the width of rectangle are determined (S903, 
S904 in Fig. 20). 

In the case where the portions corresponding to 
the table frame line-exist at the left and right edges 
of both of the partial histograms in the horizontal and 
vertical directions (namely, in the case where frame 
lines of the table exist at the left and right edges and 
the top and bottom edges of the rectangle), the cor- 
responding rectangle is determined to be a table area 
(S905, S906, S908 in Fig. 20). 

In the case where the portion corresponding to 
the table frame line lacks even at one of the left and 
right edges of both of the partial histograms in the 
horizontal and vertical directions (namely, even either 
one of the left and right edges or even either one of 
the upper and lower edges lacks), the corresponding 
rectangle is determined as a modified separator area 
(S905, S906, and S907 in Fig. 20). 

(step S205) 

Finally, the rectangle data of various kinds of 
areas which was obtained as mentioned above is gen- 
erated from the output unit 104 together with the im- 
age data. 

In the image input in step S201, by judging 
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whether the input image is a multivalue image or not 
and converting the input image into the binary image, 
the area dividing process can be executed even when 
the input image is a multivalue image such as a color 
image. 

In the image input in step S201 , when the number 
of pixels of the input image is so large that an ex- 
tremely long processing time is necessary, the proc- 
essing speed can be raised by the image t-hinning 
process for thinning out the image of (m dots in the 
vertical direction) x (n dots in the horizontal direction), 
thereby newly forming one pixel. 

In the detection of a separator or the like in step 

5203, by distinguishing the rectangle labels by a dif- 
ference between the vertical and horizontal direc- 
tions of the separator, a difference between the 
threshold values when obtaining a figure, a table, or 
the like, etc., the further detailed area attributes can 
be also classified. 

In the determination of the table area in step 

5204, as shown in Figs. 21 and 22, a difference Sx 
between the histograms at the left and right edges 
and a difference Sy between the histograms at the 
upper and lower edges are respectively obtained. 
When either one of the differences is equal to or larter 
than a threshold value Ts, it is decided that the 
lengths of rules are not uniform, so that it is deter- 
mined that the area is a modified separator area. 
When either one of the differences is less than the 
threshold value Ts, it is decided that the area is a table 
area. By such a method, they can be replaced. 

In the last output in step S206, only the image 
data of the necessary area can be also generated 
with reference to the rectangle data of various kinds 
of areas. The memory area and the processing time 
can be further reduced. 

In the above description, the histograms of the 
upper, lower, right, and left edges of the area whose 
attribute was judged as a table have been used as a 
method of determining the table area. An explanation 
will now be made with respect to an example in which 
histograms of the whole area which was determined 
to be a table are obtained and the rules are detected 
and the table area is decided on the basis of the num- 
ber of rules. 

The process in the above example is shown in a 
flowchart of Fig. 23. Processing steps similar to those 
in the flowchart of Fig. 13 described before are des- 
ignated by the same step numbers and their descrip- 
tions are omitted here, 
(step SI 204) 

Subsequently, in the table rule detector 208, his- 
tograms are calculated in the vertical and horizontal 
directions (S1501 in Fig. 26) for only the pixels (701 
in Fig. 24, 801 in Fig. 25 ) (they can be easily obtained 
by comparing and selecting the pixel label of the rec- 
tangle data and the pixel label of each pixel in the rec- 
tangle) corresponding to the table frame excluding 



the bodies or the like (702 in Fig. 24, 802 in Fig. 25 ) 
in the rectangle which was determined to be a table 
area. The threshold value Tkh of the histogram re- 
garding whether the rules are parallel with the Y axis 

5 or not from the height of rectangle and the threshold 
value Tkw of the histogram regarding whether the 
rule is parallel with X axis or notfrom the width of rec- 
tangle are obtained (81 502 and SI 503 in Fig. 26). 
The position of the threshold value Tkh or more 

10 is regarded as a rule of the table area from the shape 
of the histogram in the vertical direction (SI 504 in 
Fig. 26). The position of the threshold value Tkw or 
more is similarly regarded as a rule of the table area 
from the shape of the histogram in the horizontal di- 

15 rection (S1505 in Fig. 26), 

(step SI 205) 

The number Nx of vertical rules obtained from the 
20 histogram In the vertical direction while regarding the 
portions of the threshold value Tkh or more as rules 
is stored. The number Ny of horizontal rules obtained 
from the histogram in the horizontal direction while re- 
garding the portions of the threshold value Tkw or 
25 more as rules is stored. When three or more vertical 
rules and three or more horizontal rules exist, the 
area is determined to be a table area (S1506, SI 508, 
S1509 in Fig. 26). When the number of rules is less 
than three, the area is decided to be a modified sep- 
30 aratorarea (S1506, 81508, S1507 in Fig. 26). 

(step S1206) 

Finally, the rectangle data of various kinds of 

35 areas obtained as mentioned above is generated from 
the output unit 104 together with the image data. 

In the image input in step S201, by judging 
whether the input image is a multivalue image or not 
and by converting the input image into the binary im- 

40 age, the area dividing process can be performed 
even when the input image is a multivalue image such 
as a color image. 

In the image input in step S201 , when the number 
of pixels of the input image is so large that an ex- 

45 tremely long processing time is necessary, the proc- 
essing speed can be also raised by the image thjn- 
ning process for thinning out the image of (m dots in 
the vertical direction) x (n dots in the horizontal direc- 
tion), thereby newly forming one pixel. 

50 In the detection or the like of a separator in step 

8203, by distinguishing the rectangle labels by a dif- 
ference between the vertical and horizontal direc- 
tions of the separator, a difference between the 
threshold values when obtaining a figure, a table, or 

55 the like, etc., the further detailed area attributes can 
be also classified. 

In the table rule detection in step S1204, when it 
is known that the inclination of the image after it was 

10 
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inputted is small, by tracing the lines in place of cal- 
culating a histogram, the table rules can be also de- 
tected. 

In the table area determination in step SI 205, in 
place of fixing the threshold value TN to decide 5 
whether the area is a table area or not, as shown in 
Fig. 27, the area of rectangle is compared with a 
threshold value SI and the preset number of rules is 
compared with threshold values Tnl (area ^ SI) and 
Tn2 (area < SI). When the rules of the threshold val- io 
ues or more exist, the area is determined to be a table 
area. When the number of rules is less than the 
threshold values, the area is decided as a modified 
separator area. Due to this, they can be replaced. 

Further, as shown in Fig. 28, the threshold values is 
which were determined in correspondence to the 
magnitude of the area of the rectangle which was 
temporarily determined as a table area are decided 
on the basis of the width and height of the rectangle 
and are respectively set to W1 and H1. A threshold 20 
value of the number of vertical rules is set to TnH. A 
threshold value of the number horizontal rules is set 
to TnW. By such a method, they can be also replaced. 

In the last output in step SI 206, only the image 
data of the necessary area can be also generated 25 
with reference to the rectangle data of the various 
kinds of areas. The memory area and processing time 
can be further reduced. 

The method of obtaining the histogram of the 
area whose attribute was determined to be a table 30 
area has been used as a method of determining the 
table area in the above description. Explanation will 
now be made with respect to an example in which the 
table area is determined by the ratio of the area in the 
table frame to the area of the area which was decided 35 
to be a table area. 

The process in this example is as shown in the 
flowchart of Fig. 13 except that the process in step 
S204 differs from that in Fig. 13. Step S204, there- 
fore, will now be described. 40 

(step S204) 

Subsequently, in the table area determiner 208, 
a table area is decided. Fig. 33 is a flowchart for ex- 45 
plaining in detail the determining process of the table 
area in step S204. The determining process will now 
be described hereinbelow with reference to the flow- 
chart of Fig. 33. 

First, in step S221 , an outline of an outer frame of so 
a table is traced and point coordinates constructing 
the outline are stored. 

Fig. 34 is a diagram for explaining the processes 
in steps S221 to S223. 

In Fig. 34, reference numeral 2301 denotes a 55 
(very small) rectangle area which was determined to 
be a table; a white circle indicates a white pixel; a 
black dot denotes a black pixel; and a gray dot indi- 



cates a pixel which belongs to the inside of the table 
although it is a white pixel. 

The black pixels or the like by characters in the 
table are omitted. 

First, the inside within the table area is observed 
every line from the top and from the left side, thereby 
searching the black pixels constructing the table 
frame. When the black pixel is found out, its coordin- 
ates are stored into the memory 103. In the example 
of the area 2301 , the table area is observed in the di- 
rection from the coordinates (0, 1) to the coordinates 
(10, 1) to see if there is a black pixel exists or not ev- 
ery pixel. Since the black pixel exists at the position 
of the coordinates (2, 1), the coordinates are stored 
into the memory 103. 

Subsequently, the outline is traced clockwise or 
counterclockwise from the black pixel and the coor- 
dinates of the black pixels (hereinafter, referred to as 
outline points) constructing the outline are stored into 
the memory 1 03. In the example of the area 2301 , the 
coordinates up to (2, 1) - (10, 1), (10, 1) - (10, 11), (10. 
1 1 ) - (1 , 11 ), (0, 1 0) - (0. 4), and (1 , 4) - (1 , 2) are stored 
as outline points into the memory 103. 

In step S222, the number of pixels which belong 
to the inside of the table is counted every horizontal 
line and added. In this case, two points at the right- 
most edge and leftmost edge of the outline points 
stored in the memory 103 in step S221 are extracted 
for every line. The distance between the two extract- 
ed points is added as the number of pixels belonging 
to the inside of the table. This process can be shown 
by the following equation, 
(x coordinate of the right edge) - (x coordinate of 
the left edge) + 1 = (the number of pixels in the 
table) 

By the above method, even when the outline is 
complicated or the table frame lacks, the area in the 
table can be easily and rapidly obtained. Although 
such a process is sufficient as a determining process 
which is executed in the table area determiner 208. 
the area in the table can be also obviously accurately 
obtained. 

In the example of the area 2301 in Fig. 34, the 
line whose y coordinate is equal to 1 as the top hori- 
zontal line will now be considered. On the line of y = 
1 , the coordinates of the outline point of the leftmost 
edge are set to (2, 1). The coordinates of the outline 
point of the rightmost edge are set to (10, 1). There- 
fore, the number of pixels in the table is 
10-2+1 =9 

In step S223, a check rs made to see if the num- 
bers of pixels have been counted with respect to all 
of the lines in the table area or not. If YES, step S224 
follows. When the line in which the number of pixels 
is not yet counted still remains, the processing routine 
is returned to step S222 and the processes are con- 
tinued. 

In the example of the area 2301 , since the count- 



21 



EP 0 587 450 A1 



22 



ing operations regarding all of the lines are not yet fin- 
ished, the processing routine is returned to step S222 
and the processes are executed for the line of y = 2. 
In the line of y = 2, the coordinates of the outline point 
of the leftmost edge are set to (1, 2) and the coordin- 
ates of the outline point of the rightmost edge are set 
to (10, 2). 

Therefore, the number of pixels in the table is 
10 - 1 + 1 = 10 

This number is added to the number (9) of pixels 
calculated with respect to the line of y = 1, so that the 
total number of pixels is equal to 19. 

Similarly, in the example of the area 2301, the 
above processes are executed up to the line of y=11. 
Thus, the number of pixels in the table is 
9 + 10+10 + 11+11+11+11+11+11 
+ 11 + 10 = 116 

In step S224, a ratio of the area Sframe the frame 
to the area Stawe of the rectangle area which was de- 
termined to be a table area is obtained. 

In step S225, a check is made to see if the ratio 
of the area in the frame is larger than a threshold val- 
ue or not. Namely, when the following relation is sat- 
isfied, 

(area Sframe in the frame) -r (area Stawe of the rec- 
tangle area which was decided to be a table area) > 

(threshold value y) 
the relevant rectangle is determined to be a table area 
In step S226. The processing routine advances to 
step 8205. When the above relation is not satisfied, 
in step 8227, the relevant rectangle is decided to be 
a modified separator area and the rectangle label of 
the relevant rectangle stored in the memory 103 is 
changed to -3. After thaU step S205 follows. The 
threshold value y is set to y = 0.9 here. 
In the example of the area 2301 , 
(area in the frame) = 9+10 + 10+11 + 11 
+ 11 + 11 + 11 + 11 + 11 + 10 = 116 
(area of the rectangle area which was deter- 
mined to be a table area) =11x11 = 121 
Therefore, 116 121 = 0.959 > 0.9 and the following 
relation is satisfied, 
(area in the frame) ~ (area of the rectangle area 
which was decided to be a table area) > (threshold 
value y) 

Therefore, the area 2301 is determined to be a ta- 
ble area and step S205 follows. 

Similarly, in the example shown in Fig. 1 8, the in- 
side of the table surrounded by a frame 702 is ex- 
pressed like a black portion 801 in Fig. 19 and the ra- 
tio of the area in the frame is larger than the threshold 
value y, so that such a portion is determined to be a 
table and step S205 follows. 

In Fig. 29, reference numeral 701 denotes the 
rectangle area which was determined to be a table; 
702 the frame; and 703 a line of a horizontal writing 
sentence. 

Similarly, in the example shown in Fig. 31, the in- 



side of the table surrounded by a frame 902 is shown 
like a black portion 1001 in Fig. 32 and the ratio of the 
area in the table frame is smaller than the threshold 
value y, so that such a portion is determined to be a 

5 modified separator. 

In Fig. 31, reference numeral 901 denotes a rec- 
tangle area which was decided to be a table; 902 the 
frame; and 903 a line of a vertical writing sentence. 
In the table area determination in step S204, al- 

10 though the number of pixels belonging to the inside 
of the table has been counted every horizontal line 
and added in step 8222, it is also possible to count 
and add the number of pixels belonging to the inside 
of the table for every vertical line. 

15 In the table area determination in step 8204, al- 

though the number of pixels belonging to the inside 
of the table has been counted and added for every 
horizontal line in step 8222, the table area determin- 
ing process can be performed irrespective of the exe- 

20 cution of the operation to count the number of black 
pixels constructing the sentence in the frame in this 
instance. 

In the table area determination in step 8204, the 
number of pixels belonging to the inside of the table 

25 has been counted and added every horizontal line in 
step 8222 and the area in the frame has been calcu- 
lated. However, the table area determining process 
can be executed even when the area in the frame is 
obtained by any other means. 

30 In the last generation In step S205, only the im- 

age data of the necessary area can be also generated 
with reference to the rectangle data of various kinds 
of areas. The memory area and the processing time 
can be further reduced. 

35 For example, a process to correct the areas 

which were divided and obtained by the method as 
mentioned above and the attributes of those areas 
will now be described. 

Fig. 35 is a block diagram showing a construction 

40 of the image processing apparatus of the embodi- 
ment. 

In Fig. 35, constructions shown by reference nu- 
merals 101 to 105 are substantially the same as those 
shown In Fig. 12 and their descriptions are omitted 
45 here. 

Reference numeral 117 denotes a display unit for 
overlapping the results divided by the area divider 
1 05 to the original image data, for displaying a frame, 
and for displaying a list of attributes or the like. 
50 Reference numeral 112 denotes an area division 

corrector to correct the erroneous division of the 
areas divided by the area divider 105. 

Reference numeral 113 denotes an area attribute 
corrector to correct the area attribute which was pre- 
ss sumed by the area divider 105. 

Fig. 36 is a flowchart showing an area division 
correcting process in the image processing appara- 
tus of the embodiment. A control program to execute 
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the area division correcting process has been stored 
in the memory 103. 

Fig. 39 is a flowchart showing an area attribute 
correcting process in the image processing appara- 
tus of the embodiment. A control program to execute 
this process has been stored in the memory 103. 

(1) The case where the area which should inher- 
ently be one area was erroneously divided into a plur- 
ality of areas: 

In the division corrector 112, a plurality of areas 
which were erroneously divided by the area divider 
105 are indicated by surrounding them by a frame 
(301 in Fig. 37) by using a pointing device or the like 
(S2501 in Fig. 36). In this instance, the corresponding 
area frames (areas A, B, and C in Fig. 37) are shown 
by changing a display line from the solid lines to the 
dotted lines (S2503 in Fig. 36). 

When an area exceeding the designated frame 
exists, the designated frame is invalidated and erased 
and the processing routine is returned to the first step 
(S2502 in Fig. 36). 

When alt of the areas exist within the designated 
frame, the area data (coordinates, attribute, and the 
like in Fig. 38) of all of the areas in the frame sur- 
rounded by the frame are preserved in the memory 
103 (S2504 in Fig. 36). At the same time, a list table 
for setting new area attributes (303 in Fig. 37) is dis- 
played (S2505 in Fig. 36). 

By selecting the attributes of the new area from 
the list table (S2506 in Fig. 36), the new attributes are 
reversed and displayed (S2507 in Fig. 36). When the 
new attributes are determined (S2508 in Fig. 36), 
either one of a plurality of areas existing in the des- 
ignated frame is unconditionally selected and the co- 
ordinates, attributes, and the like of the area data are 
changed (S2509 in Fig. 36). The other remaining area 
data is invalidated (S2510 in Fig. 36). A new area 
frame (302 in Fig. 37) is displayed (S2511 in Fig. 36). 

A plurality of areas are combined to one area by 
the above operation. 

In the case where the attributes of the area are 
wrong: 

In the attribute corrector 113, in the case where 
the operator wants to change only the attributes of the 
area data which was presumed by the area divider 
105, only a target area (602 in Fig. 40) is designated 
by surrounding by a frame (603 in Fig. 40) by using a 
pointing device or the like (S2801 in Fig. 39) and a list 
table (601 in Fig. 40) of the attributes is displayed 
(S2802 in Fig. 39). In this instance, the current attri- 
butes are reversed and displayed (S2803 in Fig. 39) 
and the relevant area frame is changed from the solid 
line to the dotted line (602 in Fig. 40) (S2804 in Fig. 
39). 

By selecting the attributes to be changed from the 
list table (S2805 in Fig. 39), the reversed display of 
the original attributes is erased and the new attributes 
are reversed and displayed {S2806 in Fig. 39) and the 



attributes of the area data are also simultaneously 
changed (S2807 in Fig. 39). 

In the division corrector 112, in place of designat- 
ing the area by using the pointing device or the like, 
5 the area can be also designated by vertically or hor- 
izontally moving a cursor by using a key on a key- 
board. 

In place of showing a fact that a plurality of areas 
were selected by the designated frames by displaying 
10 using the dotted lines, such a fact can be also shown 
by changing the display color of the area frame. 

The area data structure can include not only the 
data shown in the example of Fig. 38 but also data 
such as the number of pixels in the area, writing di- 
15 rection of the sentence, or the like. 

It is also possible to first execute only the com- 
bining process of the areas and to independently per- 
form the attribute detecting process of the area later. 
In the attribute corrector 113, in place of display- 
20 ing the current attributes in the attribute list table by 
the reversing display, they can be also shown by 
changing the display color. 

In place of displaying the relevant area frame by 
changing from the solid line to the dotted line, it can 
25 be also shown by changing the frame color. 

In place of displaying the list table of attributes, by 
displaying only the current attributes and by changing 
the attribute display portion by instructing using the 
pointing device or the like or by providing a display 
30 change switch, and by repeating such a changing 
process until desired attributes are displayed, the at- 
tributes can be also changed. 

(Embodiment 3) 

35 

A preferred third embodiment of the invention will 
now be described with reference to Figs. 41 to 46. 

Fig. 41 is a block diagram showing an image proc- 
essing apparatus according to the third embodiment 
40 of the invention. 

In the diagram, reference numeral 101 denotes 
the input unit of image data. Reference numeral 102 
denotes the central processing unit (CPU) to control 
the apparatus in accordance with the control pro- 
45 grams stored in the memory 1 03 and to execute proc- 
essing arithmetic operations in each section. 

Reference numeral 103 denotes a memory to 
store control programs and various kinds of data, 
which will be explained hereinlater, are stored. 
50 Reference numeral 1 05 denotes the area divider 

to divide the area and to presume the attributes of the 
area. 

Reference numeral 117 denotes the display unit 
for overlapping the results divided by the area divider 
55 1 05 to the original data and displaying frames, there- 
by displaying a list table or the like of the attributes. 

Reference numeral 112 denotes the division col- 
lector to correct the erroneous division of the areas 
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divided by the area divider 105. 

Reference numeral 113 denotes the attribute cor- 
rector to correct the area attributes which were pre- 
sumed by the area divider 1 05. 

Reference numeral 104denotestheoutputunitto 5 
generate the results of the arithmetic operations, the 
results of the Image processes, and image data. 

Fig. 42 is a flowchart showing the area division 
correcting process in the image processing appara- 
tus of the embodiment. The control program to exe- io 
cute the above processes have been stored in the 
memory 103. 

Fig. 45 is a flowchart showing an area attribute 
correcting process in the image processing appara- 
tus of the embodiment The control program to exe- is 
cute this process has been stored in the memory 1 03. 

The correcting process in the case where an area 
which should inherently be one area was erroneously 
divided into a plurality of areas will now be described 
with reference to Fig. 42. 20 

In the division corrector 112, one of a plurality of 
areas which were erroneously divided by the area div- 
ider 105 is designated and selected as a key area by 
the pointing device or the like (S201 in Fig. 42). The 
area data (coordinates, attributes, etc.: refer to Fig. 25 
44) of the key area (refer to an area A in Fig. 43) is pre- 
served in the memory 103 (S202 in Fig. 42). 

Another divided area (refer to an area B in Fig. 
43) is designated by the pointing device or the like 
(S203 in Fig. 42). The area data (coordinates, attri- 30 
butes, etc.) of such an area is preserved into the 
memory 103 (S204 in Fig. 42). 

In this instance, a fact that those two areas were 
selected is shown by displaying a line segment (refer 
to a line segment in Fig. 43) connecting the designat- 35 
ed points in both areas (S205 in Fig. 42). A check is 
made to see if another area exists between the two 
areas or not (S206 in Fig. 42). If NO, those two areas 
are combined and the area data of the area B is in- 
validated (S207 in Fig. 42). The coordinates of the 40 
area data of the key area are updated (however, the 
attributes are not changed) (S208 in Fig. 42). Conse- 
quently, the attributes of a new area (refer to an area 
C in Fig. 43) obtained by combining the two areas are 
the same as the attributes of the key area. 45 

In the case where another area exists between 
the two areas, the combining process is stopped 
(S209 in Fig. 42). 

By repeating the above operation, a plurality of 
areas can be combined to one area. 50 

The correcting process in the case where the at- 
tributes of an area are wrong will now be described 
with reference to Fig. 45. 

In the attribute corrector 113, in the case where 
the operator wants to change only the attributes of the 55 
area data which was presumed by the area divider 
1 05, a target area (refer to 602 in Fig. 46) is continu- 
ously designated twice by the pointing device or the 



like (S501 in Fig. 45) and the list table of the attributes 
(refer to 601 in Fig. 46) is displayed (S502 in Fig. 45). 
In this instance, the current attributes are reversed 
and displayed (S503 in Fig. 45) and the relevant area 
frame is displayed by changing from the solid line to 
the dotted line (refer to 602 in Fig, 46) (S504 in Fig. 
45). 

By subsequently selecting the attributes which 
the operator wants to change from the list table (S505 
in Fig. 45), the reversed display of the original attri- 
butes is erased and the new attributes are reversed 
and displayed (S506 in Fig. 45) and the attributes of 
the area data are also simultaneously updated (S507 
in Fig. 45). 

In the division corrector 112, in place of designat- 
ing the area by the pointing device or the like, the 
area can be also designated by vertically or horizon- 
tally moving a cursor by a key on a keyboard. 

On the other hand, a fact that two areas were se- 
lected has been indicated by displaying the line seg- 
ment connecting the designated points in both areas. 
In place of such a method, however, the area frames 
of two areas can be also displayed by changing the 
display color. 

Further, the area data structure can include not 
only the data shown as an example in Fig, 44 but also 
the data such as the number of pixels in the area, writ- 
ing direction of the sentence, and the like. 

In the attribute corrector 113, in place of continu- 
ously designating the area twice by using the pointing 
device or the like, it is also possible to construct in a 
manner such that an attribute changing mode switch 
is provided and is turned on or off, thereby changing 
the meaning of the instruction of the pointing device 
or the like (specifically speaking, the ON-state of the 
attribute changing mode switch indicates the area in- 
struction to change the attributes of the area, and the 
OFF-state indicates the area instruction to correct 
the division of the area). 

With respect to the display of the current attri- 
butes in the list table of the attributes, they can be 
also displayed by changing the display color in place 
of reversing and displaying. 

Further, in place of displaying the relevant area 
frame by changing from the solid line to the dotted 
line, it can be also displayed by changing the frame 
color. 

In place of the list table of the attributes, it is also 
possible to construct in a manner such that only the 
current attributes are displayed and the display of the 
attribute display portion is changed by an instruction 
by using the pointing device or the like or by providing 
and operating a display changing switch, and by re- 
peating such an operation until desired attributes are 
displayed, the attributes can be changed. 
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Claims 

1. An image processing apparatus connprising: 

innage input means for inputting an image; 

image thinning means for getting the OR 5 
between (m dots in the vertical direction) x (n 
dots in the horizontal direction) for the Input im- 
age from said image input means, thereby thin- 
ning out (m X n) pixels and newly forming one pix- 
el; 10 

labeling means for adding a label every 
line to black pixels of the image which was thin- 
ned out by said image thinning means and adding 
the same label to the pixels which are continuous 
in the vertical, horizontal, or oblique direction and 15 
simultaneously tracing a rectangle; and 

separator detecting means for distinguish- 
ing a rectangle corresponding to a sentence por- 
tion, a rectangle corresponding to a figure or pho- 
tograph, a rectangle corresponding to a table, a 20 
rectangle corresponding to a separator, and the 
like by using a width, a height, and an area of said 
rectangle added with the label by said labeling 
means and by using the number of pixels for said 
area, namely, a pixel density, 25 

wherein an area of the input image Is div- 
ided on the basis of the result of the detection of 
said separator detecting means. 

2. An apparatus according to claim 1, further com- 30 
prising: 

writing direction detecting means for pre- 
suming a vertical writing sentence or a horizontal 
writing sentence by comparing the width and 
height of the rectangle corresponding to the sen- 35 
tence portion distinguished by said separator de- 
tecting means; 

index detecting means for detecting an in- 
dex by using the writing direction judged by said 
writing direction detecting means and a character 40 
size; and 

rectangle combining means for combining 
the rectangles by using an imaginary rectangle 
obtained by enlarging the size of inherent rectan- 
gle. 45 

3. An image processing method comprising the 
steps of: 

inputting image information; 

recognizing a rectangle by detecting con- so 
tinuous pixels for black pixels of said image infor- 
mation; 

discriminating attributes of said rectangle 
by using a width, a height, an area, and a pixel 
density of an area indicated by said recognized 55 
rectangle; and 

calculating and comparing partial histo- 
grams of said recognized area, thereby determin- 



ing the attributes of said area. 

4. An image processing method comprising the 
steps of: 

inputting image information; 

recognizing a rectangle by detecting con- 
tinuous pixels for black pixels of said image infor- 
mation; 

discriminating attributes of said rectangle 
by using a width, a height, an area, and a pixel 
density of an area indicated by said recognized 
rectangle; 

detecting rules of the recognized area; 

and 

determining attributes of said area by the 
number of rules. 

5. An image processing method comprising the 
steps of: 

inputting image information; 

recognizing a rectangle by detecting con- 
tinuous pixels for black pixels of said image Infor- 
mation; 

discriminating attributes of said rectangle 
by using a width, a height, an area, and a pixel 
density of an area indicated by said recognized 
rectangle; and 

determining attributes of said area from a 
ratio between the area of the area which was 
judged to be a table area and an area in a table 
frame. 

6. A method according to claim 3, 4, or 5, wherein 
said attributes indicate either a body or a figure 
or a table. 

7. An image processing apparatus comprising: 

input means for inputting image informa- 
tion; 

rectangle recognizing means for recogniz- 
ing a rectangle by detecting continuous pixels for 
black pixels of said image information; 

attribute discriminating means for discrim- 
inating attributes of said rectangle by using a 
width, a height, an area, and a pixel density of an 
area indicated by said recognized rectangle; and 

control means for controlling so as to de- 
termine attributes of said recognized area by cal- 
culating and comparing partial histograms of said 
area. 

8. An Image processing apparatus comprising: 

input means for inputting image informa- 
tion; 

rectangle recognizing means for recogniz- 
ing a rectangle by detecting continuous pixels for 
black pixels of said image information; 

attribute discriminating means for discrim- 
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inating attributes of said recognized rectangle by 
using a width, a height, an area, and a pixel den- 
sity of an area indicated by said rectangle; 

rule detecting means for detecting rules In 
said recognized area; and 5 

control means for controlling so as to de- 
termine attributes of said area by the number of 
said rules. 

9. An image processing apparatus comprising: io 
input means for inputting image informa- 
tion; 

rectangle recognizing means for recogniz- 
ing a rectangle by detecting continuous pixels for 
black pixels of said image information; is 

attribute discriminating means for discrim- 
inating attributes of said recognized rectangle by 
using a width, a height, an area, and a pixel den- 
sity of an area indicated by said rectangle; and 

control means for controlling so as to de- 20 
termine attributes of said area from a ratio of an 
area of the area which was judged to be a table 
area and an area in a table frame. 



image; and 

correcting means for integratedly coupling 
the divisional areas designated by said area des- 
ignating means and correcting the original divi- 
sional areas. 

14. An apparatus according to claim 13, further hav- 
ing means for correcting attributes of the desig- 
nated areas in association with the division of 
said areas. 

15. Image processing apparatus or method in which 
an input image is analysed to detect specific im- 
age configurations, and the image is divided in 
accordance with the detected configurations into 
labelled rectangle data. 



1 0. An apparatus according to claim 7, 8 or 9, wherein 25 
said attributes indicate either a body or a figure 
or a table. 



11. An image processing method comprising the 
steps of: 30 

inputting image information; 

area-dividing said inputted image informa- 
tion every attribute; 

storing area information which was area- 
divided and attribute information; 35 

instructing to correct said attribute infor- 
mation stored; and 

updating the stored attribute information in 
accordance with said instruction. 

40 

12. An image processing apparatus comprising: 

input means for inputting image informa- 
tion; 

area-dividing means for area-dividing said 
inputted image information every attribute; 45 

storing means for storing area information 
which was area-divided and attribute information; 

correction instructing means for instruct- 
ing to correct said stored attribute information; 
and 50 

updating control means for updating the 
stored attribute information in accordance with 
said instruction. 



13. An image processing apparatus comprising: 55 
area designating means for designating 
special divisional areas to be coupled among a 
plurality of divisional areas included in an input 
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